Improve screenshot capture performance (0.7.1) - #37
Conversation
Skip the full GPU readback/encode path when the capture boundary has not painted since the last accepted frame, and remove the post-capture stateAnchor short circuit that only avoided store/upload. Document JPEG frame format and the new paint_generation_unchanged diagnostic outcome. Co-authored-by: Chinmay Kabi <chinmay@blend.to>
Return JPEG bytes and SHA-256 together from compute() so hashing no longer blocks the UI isolate after every non-coalesced capture. Co-authored-by: Chinmay Kabi <chinmay@blend.to>
Replace per-frame compute() spawns with a long-lived worker that accepts TransferableTypedData, keeping JPEG encode and SHA-256 off the UI isolate without a second full-frame copy. Widget tests fall back to compute under FakeAsync. Co-authored-by: Chinmay Kabi <chinmay@blend.to>
Replace the second full-size picture.toImage mask composite with in-place RGBA fills on the encode worker, cutting one full-frame GPU raster on masked captures while preserving the dark mask fill color. Co-authored-by: Chinmay Kabi <chinmay@blend.to>
Remove the UI-isolate 9x8 thumbnail round-trip (picture.toImage + toByteData) and derive the perceptual hash from the masked RGBA buffer on the encode worker before deciding whether JPEG encoding is needed. Co-authored-by: Chinmay Kabi <chinmay@blend.to>
Treat perceptual hashes within 2 bits as visually unchanged so minor anti-alias shimmer no longer forces full JPEG encode, store, and upload. Co-authored-by: Chinmay Kabi <chinmay@blend.to>
Drop queued scroll samples and same-contentHash frames when a newer frame is enqueued, so capture bursts upload the latest observation instead of every intermediate raster. Co-authored-by: Chinmay Kabi <chinmay@blend.to>
Engage eligible-capture skipping sooner under load now that paint-generation and dHash coalesce replace the old post-capture state-signature short circuit. Co-authored-by: Chinmay Kabi <chinmay@blend.to>
Bump tugboat and tugboat_dio to 0.7.1 and document the paint-generation gate, encode-isolate path, dHash tolerance, frame supersession, and tighter screenshot budget. Co-authored-by: Chinmay Kabi <chinmay@blend.to>
Co-authored-by: Chinmay Kabi <chinmay@blend.to>
|
Cursor Agent can help with this pull request. Just |
There was a problem hiding this comment.
Pull request overview
Optimizes screenshot capture and releases Tugboat 0.7.1.
Changes:
- Adds paint-generation gating and persistent isolate encoding.
- Introduces tolerant dHash coalescing and upload supersession.
- Updates budgets, tests, documentation, and package versions.
Reviewed changes
Copilot reviewed 24 out of 24 changed files in this pull request and generated 5 comments.
Show a summary per file
| File | Description |
|---|---|
packages/tugboat/lib/src/screenshot_capturer.dart |
Integrates capture gating and worker encoding. |
packages/tugboat/lib/src/screenshot_encode.dart |
Implements masking, hashing, and JPEG encoding. |
packages/tugboat/lib/src/screenshot_encode_isolate.dart |
Adds the persistent encode worker. |
packages/tugboat/lib/src/perceptual_hash.dart |
Adds tolerant dHash comparison. |
packages/tugboat/lib/src/controller.dart |
Handles new capture outcomes and lifecycle. |
packages/tugboat/lib/src/collector_http_sink.dart |
Supersedes queued frame uploads. |
packages/tugboat/lib/src/replay_config.dart |
Lowers the screenshot budget. |
packages/tugboat/lib/src/health.dart |
Updates budget tracking defaults. |
packages/tugboat/lib/src/sdk_version.dart |
Bumps the SDK version. |
packages/tugboat/test/tugboat_replay_test.dart |
Tests dHash tolerance. |
packages/tugboat/test/replay/screenshot_fresh_paint_test.dart |
Uses deterministic inline encoding. |
packages/tugboat/test/replay/screenshot_encode_isolate_test.dart |
Tests isolate encoding and masking. |
packages/tugboat/test/replay/capture_diagnostics_test.dart |
Updates diagnostic vocabulary. |
packages/tugboat/test/collector_http_sink_test.dart |
Tests upload supersession. |
packages/tugboat/benchmark/screenshot_budget_baseline.dart |
Updates the benchmark budget. |
packages/tugboat/pubspec.yaml |
Bumps Tugboat to 0.7.1. |
packages/tugboat/example/pubspec.yaml |
Updates the example dependency. |
packages/tugboat/README.md |
Documents capture behavior and defaults. |
packages/tugboat/CHANGELOG.md |
Records the 0.7.1 release. |
packages/tugboat_dio/pubspec.yaml |
Bumps the adapter and dependency. |
packages/tugboat_dio/README.md |
Updates installation versions. |
packages/tugboat_dio/CHANGELOG.md |
Records the compatibility release. |
docs/integration/collector.md |
Documents JPEG uploads and supersession. |
docs/design/capture-and-fingerprint.md |
Updates the screenshot pipeline design. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
| commands.send( | ||
| ScreenshotEncodeIsolateCommand( | ||
| jobId: jobId, | ||
| rgba: TransferableTypedData.fromList([input.rgba]), |
| await ensureStarted(); | ||
| final commands = _commands; | ||
| if (commands == null) { | ||
| throw StateError('IsolateScreenshotEncoder failed to start'); | ||
| } | ||
| final jobId = _nextJobId++; | ||
| final completer = Completer<ScreenshotEncodeResult>(); | ||
| _pending[jobId] = completer; |
| if (paintGeneration != null) { | ||
| _lastAcceptedPaintGeneration = paintGeneration; |
| if (allowPaintGenerationSkip && | ||
| !force && | ||
| paintGeneration != null && | ||
| paintGeneration == _lastAcceptedPaintGeneration) { |
| _pendingFrames.removeWhere( | ||
| (pending) => | ||
| pending.trigger == TugboatFrameTrigger.scroll || | ||
| pending.contentHash == incoming.contentHash, | ||
| ); |
Fix isolate dispose races, commit paint generation only after frame acceptance (keyed by render-object identity), supersede in-flight uploads before retry, correct TransferableTypedData copy claims, and add paint-generation / in-flight supersession regression tests. Co-authored-by: Chinmay Kabi <chinmay@blend.to>
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 24 out of 24 changed files in this pull request and generated 2 comments.
Suppressed comments (3)
packages/tugboat/lib/src/screenshot_capturer.dart:467
- Updating
_lastDHashfor every result makes the new tolerance drift: if stored frame H0 is reused for H1 at distance 2, state advances to H1, so H2 can be distance 2 from H1 but 4 from the still-reused H0 and also be skipped; repeated changes can diverge arbitrarily. This also commits hashes from attempts later discarded as stale. Compare against and commit only the hash of an actually accepted/stored frame.
if (encoded.dHash != null) {
_lastDHash = encoded.dHash;
packages/tugboat/lib/src/screenshot_encode.dart:95
computeDHashFromRgbasamples only one source pixel for each of the 72 hash cells (perceptual_hash.dart:13-22). The removed path first rendered the full screenshot down to a filtered 9×8 thumbnail; calling this helper directly on a viewport makes changes away from those sample points invisible, so meaningful text or control updates can be coalesced. Downsample/aggregate the full image to 9×8 in the worker before comparing adjacent pixels.
final dHash = computeDHashFromRgba(rgba, input.width, input.height);
packages/tugboat/lib/src/controller.dart:2180
- Coalescing is still allowed when this request has no compatible frame. In that case either the paint-generation gate or dHash can return an empty result,
_reuseWithoutCapture/the dHash branch returnsnoCompatibleFrame, and unchanged subsequent requests repeat the same skip instead of ever creating usable evidence (for example after a boundary-transform generation change). Bypass coalescing whenever_compatibleFrameFor(context)is null.
force: force || requiresFreshPaint,
| if (allowPaintGenerationSkip && | ||
| !force && | ||
| paintGeneration != null && | ||
| identical(boundary, _lastAcceptedBoundary) && | ||
| paintGeneration == _lastAcceptedPaintGeneration) { |
| return pending.trigger == TugboatFrameTrigger.scroll || | ||
| pending.contentHash == incoming.contentHash; |
Paint-generation coalesce now uses a subtree paint signature so nested RepaintBoundary paints invalidate the gate. Collector no longer drops pending/in-flight frames by trigger or content hash, since events reference exact frame IDs with no upload alias. Co-authored-by: Chinmay Kabi <chinmay@blend.to>
Commit dHash only after controller accept/reuse, box-filter dHash cells on large buffers, bypass paint/dHash coalesce when no compatible frame exists, and guard isolate encode against dispose races. Co-authored-by: Chinmay Kabi <chinmay@blend.to>
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 25 out of 25 changed files in this pull request and generated no new comments.
Suppressed comments (4)
packages/tugboat/lib/src/screenshot_capturer.dart:475
- Do not advance the dHash baseline before the controller accepts or successfully reuses this result.
captureAttemptcan discard the result after encoding when its route/session becomes stale, and dHash coalescing can also return no compatible frame; the next attempt can then coalesce against an unaccepted image, return empty bytes, and again produce no frame. Commit the dHash alongside the paint signature only after acceptance/reuse.
height: imageHeight,
maskRects: scaledMasks,
packages/tugboat/lib/src/capture_boundary.dart:52
- This signature misses compositing-only visual updates. Retained layers such as
OpacityLayer,TransformLayer, andOffsetLayercan mutate alpha/transform/offset in place while both the layer andPictureLayer.pictureidentities remain unchanged, so the gate can reuse a stale screenshot even though the rendered pixels changed. Track the relevant layer state or a generation that also advances for composited-layer updates before using equality as a no-readback proof.
signature = Object.hash(signature, identityHashCode(layer));
if (layer is PictureLayer) {
signature = Object.hash(signature, identityHashCode(layer.picture));
}
packages/tugboat/lib/src/controller.dart:2236
- A paint-signature match does not guarantee that a compatible frame exists. For example,
_captureContextadvancesboundaryTransformGenerationwhen the boundary moves, while the retained boundary pixels/signature may remain unchanged; this branch then calls_reuseWithoutCapture, getsnoCompatibleFrame, and never retries the capture. Only enable the gate when a compatible frame is known, or retry without the gate when reuse fails.
);
}
_lastCaptureFailure = null;
if (result.skippedByPaintGeneration) {
packages/tugboat/pubspec.yaml:4
- The release bump leaves the repository's release documentation inconsistent:
docs/README.md:31,docs/integration/production-replay-acceptance.md:15, anddocs/integration/production-replay-acceptance.md:167still identify 0.7.0 as current/under test. Update those references as part of the 0.7.1 release.
version: 0.7.1
Hash composited layer offset/transform/opacity in subtree signatures, retry full capture when paint-skip reuse finds no compatible frame, and align docs to 0.7.1. Co-authored-by: Chinmay Kabi <chinmay@blend.to>
Summary
Patch release 0.7.1 that makes screenshot capture cheaper so the post-capture
stateAnchorshort circuit can be removed safely.Changes
RepaintBoundarylayers) has not painted; replacestate_signature_short_circuitwithpaint_generation_unchangedpicture.toImagebeforeFrame/afterFrameand multipart has no hash alias)Docs updated for JPEG wire format, paint-signature skip, and budget defaults.
Perf impact (expected)
Test plan
capture_diagnostics_test,screenshot_fresh_paint_test(incl. nestedRepaintBoundarygate),screenshot_encode_isolate_testflutter testsuite in CI